INFOPLAT-13349: feat(beholder): track beholder.export.* metrics per export via custom gRPC stats handler by kirqz23 · Pull Request #2251 · smartcontractkit/chainlink-common

kirqz23 · 2026-07-14T13:54:08Z

What

Adds a beholder.export.bytes and beholder.export.duration metrics to the beholder gRPC client, restoring
per-node log volume visibility and export duration (labelled by csa_public_key and signal_type logs/metrics/traces)

According to these docs, rpc.client.request.* metrics are totally deprecated without any successors in otelgrpc semconv v1.40.0
RPC semantic convention stability migration guide

Semantic conventions for RPC metrics

Additionally, new beholder.export.* metrics are not recorded on each separate message, but on the whole batch at once. In terms of beholder.export.bytes it shows real number of exported bytes only on success without cumulative number on all retries that rpc.client.request.size was producing. This was misleading in terms of how many logs/metrics are coming through the gateway and downstream consumers.

Changes

Implements beholderStatsHandler (gRPC stats.Handler) to capture outbound message sizes via OutPayload events
Adds new metric beholder.export.bytes with attributes otel_signal and csa_public_key
Adds new metric beholder.export.duration (histogram, unit s) with attributes otel_signal, csa_public_key and error
- Uses explicit second-scaled bucket boundaries (0.005–60); the SDK defaults are millisecond-scaled, so nearly every export would land in the first bucket
Groups both instruments into a shared exportMetrics struct, attached to the metric exporter once the MeterProvider exists (attachMetrics)
Wraps log and metric exporters with metering logic that:
- Uses per-call context to isolate concurrent exports
- Records bytes only on successful exports
- Records duration on both successful and failed exports, labelled error={true,false}
- Handles retries correctly (stores size, doesn't accumulate)
- Measures duration once per logical batch, covering all retry attempts and backoff
Comprehensive test coverage including concurrency and retry scenarios

Notes

Both metrics are batch-scoped: one observation per logical export, never per gRPC message or per retry attempt.
Size is captured in HandleRPC because the uncompressed proto length is only observable inside the gRPC stack. Duration is measured in the wrapper instead, so it also covers proto marshaling, connection setup and inter-attempt backoff — and is still recorded when an export fails before any RPC is issued (e.g. an already-expired context, which produces no stats events at all).

Requires

Supports

github-actions · 2026-07-14T13:54:25Z

👋 kirqz23, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

github-actions · 2026-07-14T13:55:32Z

📊 API Diff Results

No changes detected for module github.com/smartcontractkit/chainlink-common

View full report

Copilot

Pull request overview

Adds a custom metric to restore per-node visibility into OTLP log export volume by capturing the uncompressed outbound gRPC payload size and emitting it as beholder.logs.export.bytes (labelled by csa_public_key).

Changes:

Introduces a gRPC stats.Handler (sizeCaptureHandler) to capture stats.OutPayload.Length.
Wraps the OTLP logs exporter with meteredLogsExporter to increment a bytes counter on successful exports (to avoid retry inflation).
Wires the stats handler + metered exporter into NewGRPCClient’s shared log exporter connection.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
pkg/beholder/metered_exporter.go	Adds size-capture stats handler and metered exporter wrapper emitting `beholder.logs.export.bytes`.
pkg/beholder/client.go	Wires the shared capture + exporter wrapper and attaches the gRPC stats handler(s) to the log exporter dial options.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

kirqz23 · 2026-07-20T11:25:32Z

 	dialOpts := []grpc.DialOption{
 		grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts...)),
+		grpc.WithStatsHandler(&sizeCaptureHandler{capture: capture}),
 	}


gRPC builds istats.NewCombinedHandler(...) from the registered slice of multiple handlers, that is exactly the delegating handler proposed by the Copilot. This proposal is exactly what gRPC already builds for us internally. The downside of having multiple stats handlers however is that gRPC fires every handler on every event, in registration order. So for one OutPayload event there will be two HandleRPC(...) calls which might be an overhead. We might consider either keeping both statsHandlers if we need them, or dropping grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts...)).

This old handler:

Emits rpc.client.* metrics (which part of them are going to be removed),

Creates a trace span per RPC

Injects trace-context headers

So, we have two approaches here:

Keep two stats handlers as we have it now. exportSizeHandler shouldn't add much to the performance.

Drop grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelOpts...)) taking into account that we loose 3 points mentioned in the above comment, however rpc.client.* metrics are totally deprecated anyway expect duration, which can be added to our custom handler alongside beholder.export.bytes, e.g. sth like beholder.export.duration

cc @pkcll

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/beholder/client.go:580

In newMeterProvider, the previous implementation appended cfg.metricOptions() (which includes sdkmetric.WithCardinalityLimit(cfg.MetricCardinalityLimit)) but the new code constructs the MeterProvider without it. This silently drops the configured per-instrument cardinality limit, which can increase time-series cardinality and memory usage in production.

	return sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metered, readerOpts...)),
		sdkmetric.WithResource(resource),
		sdkmetric.WithView(cfg.MetricViews...),
	), metered, nil

pkg/beholder/client.go:153

The PR description states beholder.export.* should be labelled per signal type (logs/metrics/traces), but the implementation only meters log and metric exports. Trace exports created in newTracerProvider are not wrapped and do not use exportSizeHandler, so beholder.export.bytes/beholder.export.duration won't be emitted for otel_signal="traces".

	// Shared export instruments beholder.export.bytes and
	// beholder.export.duration, labelled per signal. They live on this
	// MeterProvider, so the metrics exporter can only be wired up once the
	// provider and its meter exist.
	expMetrics, err := newExportMetrics(meter)

…RPC stats handler

Copilot AI review requested due to automatic review settings July 14, 2026 13:54

kirqz23 requested a review from a team as a code owner July 14, 2026 13:54

product-security-plaid-production Bot requested a review from fouadkada July 14, 2026 13:54

Copilot started reviewing on behalf of kirqz23 July 14, 2026 13:54 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

pkcll previously approved these changes Jul 15, 2026

View reviewed changes

kirqz23 dismissed pkcll’s stale review via ab408fb July 16, 2026 13:22

kirqz23 requested a review from Copilot July 20, 2026 12:42

Copilot started reviewing on behalf of kirqz23 July 20, 2026 12:43 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Comment thread pkg/beholder/client.go Outdated

Comment thread pkg/beholder/metered_exporter.go Outdated

kirqz23 requested a review from pkcll July 20, 2026 19:26

kirqz23 changed the title ~~INFOPLAT-13349: feat(beholder): track log export bytes per node via gRPC stats handler~~ INFOPLAT-13349: feat(beholder): track log export bytes per export via gRPC stats handler Jul 20, 2026

kirqz23 changed the title ~~INFOPLAT-13349: feat(beholder): track log export bytes per export via gRPC stats handler~~ INFOPLAT-13349: feat(beholder): track export bytes per export via gRPC stats handler Jul 20, 2026

kirqz23 changed the title ~~INFOPLAT-13349: feat(beholder): track export bytes per export via gRPC stats handler~~ INFOPLAT-13349: feat(beholder): track bytes per export via custom gRPC stats handler Jul 20, 2026

kirqz23 changed the title ~~INFOPLAT-13349: feat(beholder): track bytes per export via custom gRPC stats handler~~ INFOPLAT-13349: feat(beholder): track beholder.export.* metrics per export via custom gRPC stats handler Aug 6, 2026

kirqz23 force-pushed the infoplat-13349-metered-exporter branch from 1440cca to 1fec9f9 Compare August 6, 2026 09:47

Copilot AI review requested due to automatic review settings August 6, 2026 09:47

Copilot started reviewing on behalf of kirqz23 August 6, 2026 09:48 View session

Copilot AI reviewed Aug 6, 2026

View reviewed changes

feat(beholder): track beholder.export.* metrics per node via custom g…

eb7a19b

…RPC stats handler

kirqz23 force-pushed the infoplat-13349-metered-exporter branch from 1fec9f9 to eb7a19b Compare August 6, 2026 17:11

Navigation Menu

Uh oh!

INFOPLAT-13349: feat(beholder): track beholder.export.* metrics per export via custom gRPC stats handler - #2251

INFOPLAT-13349: feat(beholder): track beholder.export.* metrics per export via custom gRPC stats handler#2251
kirqz23 wants to merge 1 commit into
mainfrom
infoplat-13349-metered-exporter

kirqz23 commented Jul 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

kirqz23 Jul 20, 2026 •

edited

Loading

Uh oh!

kirqz23 Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

kirqz23 commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Changes

Notes

Requires

Supports

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 API Diff Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

kirqz23 Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kirqz23 Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kirqz23 commented Jul 14, 2026 •

edited

Loading

github-actions Bot commented Jul 14, 2026 •

edited

Loading

kirqz23 Jul 20, 2026 •

edited

Loading